Skip to main content

Ohio RiverFlows API — TLS verification on Windows Python workers

This note documents a production issue seen when the RSMS worker (or any Python process using requests) calls the Ohio RiverFlows REST API at www.ohioriver.org, and the recommended fix when the host presents a chain that Mozilla’s CA bundle (via certifi) does not already trust.

Related code: rsms-worker-function/nfq_from_api.py — _requests_verify() sets session.verify to certifi.where() by default (not the Windows certificate store).

Symptoms​

  • requests.exceptions.SSLError / ssl.SSLCertVerificationError, often with text like certificate verify failed or unable to get local issuer certificate, on HTTPS calls to RiverFlows (e.g. get-rivers, multi-riverflows-duration).
  • Browsers or other tools on the same VM may succeed because they use the Windows trust store; Python + requests + certifi use only the PEM bundle shipped with certifi.

Optional related symptom (environment-specific):

  • ohioriver.org (no www) may fail DNS resolution (getaddrinfo) while www.ohioriver.org resolves. Prefer https://www.ohioriver.org/api in OHIO_RIVER_API_BASE_URL (this matches the worker default in code and examples).

When operations have a PEM file that contains the missing CA certificate(s) for the RiverFlows site (often named like ohioriver_org.ca-bundle — a chain of -----BEGIN CERTIFICATE----- blocks, no private keys):

  1. Activate the same Python environment the worker uses (e.g. conda/venv rsms).

  2. Locate certifi’s bundle:

    python -c "import certifi; print(certifi.where())"

    Example output:

    C:\Users\gqc\Envs\rsms\lib\site-packages\certifi\cacert.pem

  3. Back up that file (e.g. copy to cacert.pem.bak in the same directory).

  4. Append the entire contents of ohioriver_org.ca-bundle to the end of cacert.pem (additional PEM blocks are valid; order of unrelated CAs does not matter for trust).

  5. Restart the worker process and re-run a RiverFlows call.

Security: Only append public certificate material. Do not copy or merge private keys (e.g. *.key) into cacert.pem or commit them to the repo.

After Python / certifi upgrades​

Running pip install --upgrade certifi (or recreating the virtual environment) can replace cacert.pem. Keep a copy of ohioriver_org.ca-bundle (or your internal ops doc) and re-append after upgrades, or automate that step in VM provisioning.

Emergency workaround (not for production)​

The worker supports OHIO_RIVER_SKIP_SSL_VERIFY (1 / true / yes) to disable TLS verification for RiverFlows only. This is insecure (vulnerable to interception) and should be used only for short-lived debugging, never as the long-term fix. Prefer appending the CA bundle as above.

Verification​

After the append:

  • From the worker environment, a small requests.get to the configured OHIO_RIVER_API_BASE_URL paths with a valid API key should return HTTP 200 (or an expected API error), not an SSL verification failure.

If verification still fails, confirm the PEM file includes all required intermediate/root CAs for the server chain; operations may need an updated bundle from whoever manages the ohioriver.org TLS configuration.

Last updated on by jacobgqc